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| Статический анализ - это... 


"..это проверка кодовой базы с помощью 
специального ПО 


"..это code review в автоматическом режиме 
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| Что может обнаружить статанализ 


" Отклонения OT code style 
" Code smells 


" Нарушения стандарта кодирования (MISRA, 
AUTOSAR С++ и т.д.) 


= Опечатки, баги, copy-paste 
" Потенциальнье уязвимости (СУУЕ, СЕКТ) 
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| Почему это необходимо? 


30х 
Относительная цена исправления ошибки 


н в зависимости от времени её обнаружения 


20х 
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| Почему это необходимо? 


30х 


Относительная цена исправления ошибки 


25х в зависимости от времени ее обнаружения 


20х 


15х 


Статанализ— 


\ 


10х 


Проектирование\ Разработка Компонентное/ | Системное/ Пост-релиз 
интеграционное | приёмочное 
тестирование тестирование 
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| Современные статические анализаторы 


e PVS-Studio • БОЛЬШОЙ СПИСОК 
* ReSharper статических анализаторов: 
• Сомегіїу 


• SonarQube 

• Klocwork 

• Clang Static Analyzer 
• IntelliJ IDEA 
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| Про интеграцию вообще 


1. Классический сценарий интеграции 


2. Проблема интеграции в открытые проекты... 
з. .иее решение 
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| Классический сценарий интеграции 
Основные принципы: 


" Анализ локально 
" Анализ на сервере 


ш (HL) HighLoad " 


is | «зд З ~“ 5 - 
(плагины для IDE, системы мониторинга 
компиляции и т.д.) нано 


| Автоматический анализ на сервере 


О, 


circleci 


(command-line-yTunutbl, плагины для С!-систем, 


, системы мониторинга) gh ca 


| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Классический сценарий интеграции 
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| Применение анализа в Open-source 


В ЧЕМ ОТЛИЧИЕ? 
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| Интеграция в открытые проекты 
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| Интеграция в открытые проекты 
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| Интеграция в открытые проекты 
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| Интеграция в открытые проекты 
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| Интеграция в открытые проекты 
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Load 


2020 


| Анализ коммитов 


" Вариант 1: инкрементальный анализ. 


" Вариант 2: прямое сравнение логов. 
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| С чего все началось 


Бесплатная лицензия PVS-Studio 
для открьтьх проектов: 


www.viva64.com/pvs-free- 
opensource 
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(нг) ноћ саа” 


Специфичные моменты при анализе 


ClickHouse 
= Собирается на Ubuntu 


= Имеет сообщество 
= Хочется наглядности 
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| Что решено было делать 


" Для анализа Ha Linux – трассировка 
компиляции 


" Для коммитов - прямое сравнение логов 
(вариант 2) 


" Для наглядности результатов - загрузка 
логов на GitHub 
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Что решено было делать 
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| трассировка компиляции ЇЇ 


" Очень просто 


« Использует strace out 


" Не зависит от сборочной системы 


= pvs-studio-analyzer trace -- 


= pvs-studio-analyzer analyze [OPTIONS | 
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| Конвертация и сохранение логов 


= plog-converter 
" Публичный бакет 53 
= GitHub АРІ 
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С) Add an SQL function formatRes- X — + 


– > с 8 github.com/ClickHouse/ClickHouse/pull/15497/commits 


Add ап SQL function formatReadableTimeDelta to format time delta 215497 


| | р = 
filipecaixeta wants to merge 5 commits into ClickHouse:master from Filipecaixeta:master |) 


add maximum unit arg to formatReadableTimaNalts functinn 


© filipecaixeta committed 6 дауз адо Х 


fix style апа рм5 check 


© filipecaixeta committed 6 days ago X 
x 


fix style and pvs check 
© filipecaixeta committed 6 days ago X x 


© Commits on Oct 8, 2020 и 
ө 
fix ру check 
© filipecaixeta committed 14 hoursago X  & 
a 


© 2020 GitHub, Inc. Terms Privacy 
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Some checks were not successful 
2 failing, 5 pending, and 8 successful checks 


PVS check — Found 2 new errors, total 30 errors 
Style check — Found 2 errors 

ClickHouse build check Pending — 7/16 builds fini... 
ClickHouse special build check Pending — 4/5 bui... 


~ Functional stateful tests (debug) Pending — Started 


г Eunctinnal ctatalace tacte (Лаћи ит Dondina — Star 


Security Status Help Contact GitHub 


Details 


Details 


Details 


Details 


Details 


Nataile 


Pricing 


» 


АР! 


С 


С 


С 


С 


Training 


ж 2 
1467973 <> 
bab7c76 €» 
0436e0f €» 
3861a03 <} 
Blog About 
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С) Add an SQL function formatRes- X — + 


– > с 8 github.com/ClickHouse/ClickHouse/pull/15497/commits 


Add ап SQL function formatReadableTimeDelta to format time delta 215497 


| | р = 
filipecaixeta wants to merge 5 commits into ClickHouse:master from Filipecaixeta:master |) 


add maximum unit arg to formatReadableTimaNalts functinn 


© filipecaixeta committed 6 дауз адо Х 


fix style апа рм5 check 


© filipecaixeta committed 6 days ago X 
x 


fix style and pvs check 
© filipecaixeta committed 6 days ago X x 


© Commits on Oct 8, 2020 и 
ө 
fix pvs check 
© filipecaixeta committed 14 hoursago X  & 
- 


© 2020 GitHub, Inc. Terms Privacy 


43 


Some checks were not successful 
2 failing, 5 pending, and 8 successful checks 


PVS check — Found 2 new errors, total 30 errors 
Style check — Found 2 errors 

ClickHouse build check Pending — 7/16 builds fini... 
ClickHouse special build check Pending — 4/5 bui... 


^ Functional stateful tests (debug) Pending — Started 


г Eunctinnal ctatalacc tacte (Лаћи ит Dondina — Star 


Security Status Help Contact GitHub 


Details 
Details 
Details 


Details 


Nataile 


Pricing 


> 


API 


С 


С 


С 


С 


Training 


» о 
1467973 <> 
bab7c76 €» 
8436е8+ с» 
3861a03 «> 
Віод About 
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я = О х 
С) Add ап SQL function formatReac X (9) ClickHouse Pvs Check for PR#15 X + 


а а clickhouse-test-reports.s3.yandex.net/15497/146f973437b0a9d62c047efee1fd583f823501e8/pvs_check.html#fail1 а * » 2 


ClickHouse Pvs Check for PR #15497 


test_run.txt.out.log PR #15497 Commit Help Task (private network) 


Test name Test status Test time, sec. 
HTML report Look at the report 
/repo. folder/src/Functions/FunctionsFormatting.h:386 | FAIL 


/repo. folder/src/Functions/FunctionsFormatting.h:459 | FAIL 


44 


(нг) HighLoad 


| = О х 
С) Add ап SQL function formatReac X (9) ClickHouse Pvs Check for PR#15 X + 


С а clickhouse-test-reports.s3.yandex.net/15497/146f973437b0a9d62c047efee1fd583f823501e8/pvs_check.html#fail1 а й $ o 
ClickHouse Pvs Check for PR #15497 
test runtxtoutlog РВ 815497 Commit Help Task (private network) 
Test name Test status Test time, sec. 
HTML report Look at the report 


/repo. folder/src/Functions/FunctionsFormatting.h:386 | FAIL 


/repo. folder/src/Functions/FunctionsFormatting.h:459 | FAIL 
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С) Add ап SQL function formatReac X @ ClickHouse Pvs Check for PR # х @ PVS-Studio HTML Report x + 


(> 8 clickhouse-test-reports.s3.yandex.net/15497/146f973437b0a9d62c047efee1fd583f823501e8/pvs_studio_report/pvs-studio-h.. © ў я 25 


Location 


Consider checking for misprints. It's possible that the 
‘response’ should be checked here. 


| General Analysis XDBCDictionarySource.cpp:221 Medium  М1051 


Calling the 'nextImpl' virtual function in the constructor 


| General Analysis WriteBufferFromArena.h:55 High V1053 may lead to unexpected result at runtime. 
| General Analysis wide integer impl.h:35 Medium  V1061 Extending the 'std' namespace may result in undefined 
behavior. 
| General Analysis Volnitsky.h:280 Medium №1048 Тһе 'chars.c1' variable was assigned the same value. 
General Analysis vdso.c:44 Medium V707 Giving short names to global variables is considered to be 


=== bad practice. It is suggested to rename ‘eh’ variable. 


A part of conditional expression is always true: first. octet 


General Analysis UTF8Helpers.h:45 Medium V560 < 0x80. The value range of char type: [-128, 127]. 
| А part of conditional expression is always false: 
| General Analysis UTF8Helpers.h:45 Medium V560 first octet >= OxF8. The value range of char type: [-128, 
127]. 
General Analysis Types.h:263 Medium 1061 Extending the "514" namespace may result in undefined 
— behavior. 
Not all members of a class are initialized inside the 
General Analysis TwoLevelHashTable.h:169 Medium №730 constructor. Consider inspecting: container, bucket, 


current it. v 
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С) Add an SQL function formatReac X | @ ClickHouse Pvs Check for РА #15 X — (V PVS-Studio HTML Report x + 


С 8 clickhouse-test-reports.s3.yandex.net/15497/146f1973437b0a9d62c047efee1fd583f823501e8/pvs studio report/pvs-studio-h.. Q ў 5o 


The variable 'prev', captured іп a lambda expression, has a 


General Analysis gtest compressionCodec.cpp:1132 Medium V788 Марианной ру 


There might Бе dereferencing of а potential null pointer 


General Analysis gtest cascade and memory write buffer.cpp:47 Medium БОЕ readable". 


General Analysis GroupByFunctionKeysVisitor.h: 100 Medium Expression "Кеер key' is always true. 
General Analysis greatCircleDistance.cpp:158 Medium Potentially uninitialized variable 'k lon' used. 
General Analysis greatCircleDistance.cpp:158 Medium Potentially uninitialized variable 'k lat' used. 


Consider checking for misprints. It's possible that the 


General Analysis. getLeastSupertype.cpp: 388 Medium "піп, bit width of integer' should be checked here. 


Dereferencing of the null pointer 


General Analysis FunctionsFormatting.h:459 High ‘maximum unit. const. сої" might take place. 


General Analysis FunctionsFormatting.h:386 High Expression 'maximum, unit int < 1' is always false. 


Uninitialized variable 'to nested type' will be used in the 


General Analysis FunctionsConversion.h: 2088 High lambda expression, as it was captured by value. 


Uninitialized variable 'from nested type' will be used in 


General Analysis FunctionsConversion.h:2088 High the lambda expression, as it was captured by value. 


Transformation of constant value 128. The value range of 


General Analysis FunctionsCoding.h:1712 Medium signed char type: [-128, 127]. 


Not all members of a class are initialized inside the 


General Analysis FixedHashTable.h:232 Medium ката Рейн 
constructor. Consider inspecting: size. 
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о Add ап SQL function formatReac X | @ ClickHouse Pvs Check for РЕ #15 X («З PVS-Studio HTML Report x + 


С 8 clickhouse-test-reports.s3.yandex.net/15497/146f1973437b0a9d62c047efee1fd583f823501e8/pvs studio report/pvs-studio-h.. Q ў 5o 


The variable 'prev', captured іп a lambda expression, has a 


General Analysis gtest compressionCodec.cpp:1132 Medium V788 Марианной ру 


There might Бе dereferencing of а potential null pointer 
"wbuf. readable'. 


General Analysis gtest cascade and memory write buffer.cpp:47 Medium 
General Analysis GroupByFunctionKeysVisitor.h: 100 Medium Expression "Кеер key' is always true. 

General Analysis greatCircleDistance.cpp:158 Medium Potentially uninitialized variable 'k lon' used. 
General Analysis greatCircleDistance.cpp:158 Medium Potentially uninitialized variable 'k lat' used. 


Consider checking for misprints. It's possible that the 
"піп, bit width of integer' should be checked here. 


General Analysis | FunctionsFormatting.h:459 High Dereferencing of the пий poer 
maximum, unit const col' might take place. 


General Analysis FunctionsFormatting.h:386 High Expression 'maximum, unit int < 1' is always false. 


General Analysis getLeastSupertype.cpp:388 Medium 


Uninitialized variable 'to nested type' will be used in the 


General Analysis FunctionsConversion.h:2088 High lambda expression, as it was captured by value. 


Uninitialized variable 'from nested type' will be used in 


General Analysis FunctionsConversion.h:2088 High the lambda expression, as it was captured by value. 


Transformation of constant value 128. The value range of 


General Analysis FunctionsCoding.h:1712 Medium signed char type: [-128, 127]. 


Not all members of a class are initialized inside the 


General Analysis FixedHashTable.h:232 Medium ката Рейн 
constructor. Consider inspecting: size. 
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С) Add ап SQL function formatReac Х | @ ClickHouse Pvs Check for PR#15 X | @ PVS-Studio HTML Report X @ FunctionsFormatting.h x T 


гу 8 clickhouse-test-reports.s3.yandex.net/15497/146f1973437b0a9d62c047efee1fd583f823501e8/pvs studio report/pvs-studio-h.. Q yr 5 2 


e «typename T» 

ecuteType(Block & block, const ColumnNumbers & arguments, size t result) const 
ing maximum unit = ""; 
(arguments.size() -- 2) 


const ColumnPtr & maximum unit column = block.getByPosition(arguments[1]).column; 
if (const ColumnConst * maximum unit const col = checkAndGetColumnConstStringOrFixedString(maximum unit colum 
maximum unit = maximum unit const col-»getValue«String»(); 
else 
throw Exception( 
"Illegal column 


+ maximum unit const col-»getName() + " of argument of function " + getName(), Errc 


| vs22 Dereferencing of the null pointer 'maximum unit const col' might take place. 


ш (нг) HighLoad 


| Итого 


= Коммиты теперь проверяются 
автоматически 


" Для каждой проверки можно посмотреть 
отдельный fullhtmi-nor 
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“Анализ заработал 


сразу как надо 
ОЧИ ја а – Алексеи 
ў Миловидов 


AS e 


» 
~ 


Load 


Еще об обеспечении качества 


= -Wall " Тесты с Address Sanitizer 

" -М/ехіга : Memory Sanitizer 

" -Weverything = Thread Sanitizer 

« Сборка одновременно и * Undefined Behaviour 
gcc u clang Sanitizer 

= Сборка с clang-tidy, " Тестовые среды с 
проверки clang static реальньми данными 
analyzer = Фаззинг 


B 
m (HL) HighLoad 
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[примерћ 


if (pos + 5 <= end 
&& pos[@] >= ӨхСӨ 88 ро5|9| <= ӨхрЕ 
&& pos[1] >= 0x80 88 pos[1] <= ОХВЕ 
&& pos[2] >= дхгд 88 pos[2] < дхвд 
&& !isAlphaASCII(pos[2]) 
&& pos[3] >= ӨхСӨ && pos[0] <= OxDF 
&& pos[4] >= 0x80 && pos[4] <= ОХВЕ) 
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[примере 


1# (ро5 + 5 <= епа 
&& pos[@] >= ӨхСӨ && pos[@] <= OxDF 
&& pos[1] >= 9х80 88 pos[1] <= ОХВЕ 
&& pos[2] >= дхгд 88 pos[2] < дхвд 
&& !isAlphaASCII(pos[ 21) 
&& pos[3] >= дхсд && роѕ[0] <= OxDF 
&& pos[4] >= 0x80 88 pos[4] <= ОХВЕ) 


V501 There are identical sub-expressions 'pos[0] <= 
OxDF' to the left and to the right of the '&&' 
operator. 

= (HL) HighLoad " 


[примере 


1# (ро5 + 5 <= епа 
&& pos[@] >= ӨхСӨ 88 pos[@] <= OxDF 
&& pos[1] >= 9х80 88 pos[1] <= ОХВЕ 
&& pos[2] >= дхгд 88 pos[2] < дхвд 
&& lisAlphaASCII(pos[2]) 
&& pos[3] >= ӨхСӨ 88 pos[3] <= OxDF 
&& pos[4] >= 0x80 88 pos[4] <= ОХВЕ) 


V501 There are identical sub-expressions 'pos[0] <= 
OxDF' to the left and to the right of the '&&' 
operator. 
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| Пример ме? ________ 


if (unlikely(array size) < 0) 
throw Exception(....); 
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| Пример м2 


if (unlikely(array size) < 0) 
throw Exception(....); 


№562 It's odd to compare а bool type 
value with а value of О: (!!(аггау 5і2е)) < О. 
V547 Expression (И(аггау size)) < O' is 
always false. 
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| Пример м2 


if (unlikely(array size < д)) 
throw Exception(....); 


№562 It's odd to compare а bool type 
value with а value of О: (!!(аггау 5і2е)) < О. 
V547 Expression (И(аггау size)) < O' is 
always false. 
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[newwepNe3 


const iterator& operator ++(int) 4 
const iterator tmp = *this; 
++*this; 
return tmp; 
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[примермез 


const_iterator& operator ++(int) 1 
const iterator tmp = *this; 
++*this; 
return tmp; 


№558 Function returns the reference to 
temporary local object: tmp. 
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[newwepNe3 


const iterator operator ++(int) 1 
const iterator tmp = *this; 
++ 115; 
return tmp; 


V558 Function returns the reference to 
temporary local object: tmp. 
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{пример No E 


if (!data type-»canBePromoted()) 
throw new Exception | .... |; 
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[пвимер nea 


if (!data type-»canBePromoted()) 
throw new Exception { .... |; 


V1022. An exception was thrown by pointer. 
Consider throwing it by value instead. 
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[newwepNe4 


if (!data type-»canBePromoted()) 
throw Exception | .... |; 


V1022. An exception was thrown by pointer. 
Consider throwing it by value instead. 
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[newwepNeS 


if (maximum unit const col) 


else 
throw Exception("Illegal column 
+ maximum unit const col-»getName() 
+ " of argument of function 
+ getName() 


, ErrorCodes::ILLEGAL COLUMN); 
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[newwepNeS 


if (maximum unit const col) 


else 
throw Exception("Illegal column 
+ maximum unit const col-»getName() 
+ " of argument of function 
+ getName() 


, ErrorCodes::ILLEGAL COLUMN); 


V522 Dereferencing of the null pointer 
maximum unit const со! might take place. 
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[newwepNeS 


if (maximum unit const col) 


else 
throw Exception("Illegal column of 
argument of function " 
+ getName() 
, ErrorCodes::ILLEGAL COLUMN); 


V522 Dereferencing of the null pointer 
maximum unit const со! might take place. 
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[newwepNeS 


if (maximum unit const col) 


V522 Dereferencing of the null pointer 
maximum unit const со! might take place. 
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= О x 
С) Add an SQL function formatRes- x +} 


< >С 8 github.com/ClickHouse/ClickHouse/pull/15497/commits/3861a0392736b866ad77e70da39aeba8c985242b © ў з 2 


Ада ап SQL function formatReadableTimeDelta to format time delta #15497 
fe Merged 


Changes from 1 commit у File filter... у X Clear filters тро... У #8 ~ 


v 4 ШИМИ  src/Functions/FunctionsFormatting.h (7) 


А @@ -455,9 +455,6 00 class FunctionFormatReadableTimeDelta : public IFunction 


const ColumnConst * maximum unit const col = checkAndGetColumnConstStringOrFixedString(maximum unit column.get()); 
if (maximum unit const col) 
maximum unit = maximum unit const col-»getValue«String»(); 
- else 
- throw Exception( 


- "Illegal column " + maximum_unit_const_col->getName() + " of argument of function " + де ате(), ErrorCodes::IL 


if (const ColumnVector«T» * col from = checkAndGetColumn«ColumnVector«T»»(block.getByPosition(arguments[9]).column.get())) 


00 -484,7 «481,6 @@ class FunctionFormatReadableTimeDelta : public IFunction 


181 block.getByPosition(result).column = std::move(col to); 


return true; 
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| Заключение 


Настроить статанализ в open-source-npoekre 
- зто не больно и не страшно! 
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| Подарок для слушателей от PVS-Studio 


Бесплатная лицензия для Промокод на месяц: 
open-source: 


www.Viva64.com/pvs-free- www.viva64.com/pvs-download- 
„  Opensource highload 
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